Modeling Conditional Independence with Nonuniform Predictions

A statistical language model assigns probability to strings of arbitrary length. Unfortunately, it is not possible to gather reliable statistics on strings of arbitrary length from a nite corpus. Therefore, a statistical language model must decide that each symbol in a string depends on at most a small, nite number of other symbols in the string. In this abstract, we propose a new way to model conditional independence in Markov models. The central feature of our nonuniform Markov model is that it makes predictions of varying lengths using contexts of varying lengths. We believe that our work has two contributions to o er to the eld of Markov modeling. The rst contribution is our interpretation of the interpolation parameters of an interpolated Markov model (Jelinek & Mercer 1980) as beliefs about conditional independence. Prior work has interpreted the interpolation parameters as hidden state transitions between models of different orders, or as smoothing \speci c probabilities" with \general probabilities". Our interpretation gives rise to the second contribution of our work, namely, the use of nonuniform predictions. Nonuniform predictions is a principled way to perform alphabet extension, that is, to make a string become a symbol in the alphabet, an ad hoc technique that can improve model performance (Jeanrenaud et al. 1995). This paper consists of three sections. First, we provide three di erent generative intepretations for the interpolation parameters of a Markov model. These interpretations give rise to a state model, a context model, and our nouniform model. Next, we compare the ability of these three interpretations to model local independence and global independence. We argue that the nonuniform model combines the ability of the state model to properly model global independence with the ability of the context model to properly model local independence. Finally, we prove that the nonuniform interpretation is fundamentally di erent from either uniform interpretation because it is not possible to map a nonuniformmodel into an extensionally equivalent uniform model. The full paper (Ristad & Thomas 1996) provides e cient algorithms for evaluating the probability of a string according to a nonuniform model, for nding the most likely nonuniform generation path for a given string, and for optimizing the parameters of a nonuniform model on a training corpus. Our notation is as follows. Let A be a nite alphabet of distinct symbols, jAj = k, and let x 2 A denote an arbitrary string of length T over the alphabet A. Then xji denotes the substring of x T that begins at position i and ends at position j. For convenience, we abbreviate the unit length substring xi as xi and the length t pre x of x as x.